In digital image capture systems such as digital cameras, an image sensor captures image data, and the image data are processed by a sequence of image processing procedures arranged in an imaging pipeline to give the desired output color images. Typical image sensors use a color filter array arranged in a mosaic pattern to capture one primary color (R, G or B) per pixel. An example 16×16 Bayer color filter array 10 is shown in FIG. 1 of the drawings. For typical image sensors consisting of millions of pixels, one of the four 2×2 basic units or blocks of pixels are possible, as is shown in FIG. 2 of the drawings, is replicated throughout the entire sensor, where the basic blocks are indicated by reference numerals 12 through 18. The symbols G1 and G2, both representing green, are used for the purpose of specifying their relative location within the 2×2 basic block.
A demosaicing procedure is used in the imaging pipeline to convert the mosaic data into full color image data where complete RGB values are specified at each pixel location. Generally, the steps before the demosaicing step are referred to as being in the Bayer domain, and the steps after the demosaicing step are referred to as being in the RGB domain.
In many imaging applications, the actual required output size of the image could vary. Consider an example of a digital camera with a sensor of 1600×1200 pixels. When the camera operates in normal capture mode, the full size image is captured and sent to a storage device, or to an external system such as an image printer or a computer. When the camera operates in the preview mode, an image is processed and sent to a preview screen to be displayed, which could have a size of, e.g. 320×240 pixels. In this case, the image will need to be spatially scaled down at some point within the imaging pipeline.
Many image sensors support a decimation mode where rows and columns of the sensor data are skipped in the read-out process so that the image read from the sensor is only at a fraction of the sensor resolution or size. A disadvantage of using sensor decimation is that this process usually affects the frame rate, i.e. the number of images per unit time. When a user wants to capture an image, the system is switched from preview to capture mode. In this case, it would be necessary to set the sensor read-out mode back to full resolution. This causes a change in frame rate, and as a result the sensor exposure parameters would have to be adjusted to account for the frame rate change. This can cause a delay in image capture because image sensors take time to settle down when the sensor exposure parameters are changed.
An alternative approach is to keep the sensor read-out resolution constant, and scale down the input image at the front end of the imaging pipeline, i.e. perform scale down immediately after the image data are received at the imaging pipeline. This approach maintains a constant frame rate and hence significantly simplifies exposure control and delay in the system. Similar to the sensor decimation approach, the image size is reduced at the front end of the pipeline and hence power consumption of the image processor can be minimized.
One technique for scaling down an image in the Bayer domain is decimation or sub-sampling. FIG. 3 shows an example of decimation by a factor of 4. Referring to FIG. 3, an input Bayer domain image 20 having a resolution of 16×16 is scaled down to produce an output image containing 4×4 pixels. To produce the image 22, the image 20 is processed horizontally (from left to right) such that pixels from two consecutive columns are retained, and the next 6 are dropped, and so on. Similarly, the image 20 is processed in the vertical direction (from top to bottom) such that pixels from two consecutive rows are retained, the next 6 rows are dropped, and so on. The result is the image 22 where each of the horizontal and vertical dimensions is reduced by a factor of 4, with the output image having the exact same mosaic arrangement as the input image 20.
In the general case of decimation by 2K (vertical) and 2L (horizontal) for integers K and L (K and L do not need to have the same value), the procedure follows an alternating pattern of retaining pixels in the first two rows, skip the next 2K+1−2 rows, retain the next two rows, skip the next 2K+1−2 rows, and so on. A similar procedure is performed in the horizontal direction where one follows an alternating procedure of keeping 2 columns and dropping the next 2L+1−2 columns.
Another technique for decimation includes smoothing or averaging. Consider the case of decimation with factors 2K (vertical) and 2L (horizontal). In this case, the image is partitioned into tiles of size 2K+1 and 2L+1. Each output pixel in the output image is given by an average of all the pixels of the same color within a 2K+1 and 2L+1 tile. FIG. 4 shows an example where K=L=2. Here Ravg is the weighted average of the 16 R pixels, G1avg is the weighted average of 16 G1 pixels, and so on. The size of the kernel (the number of pixels included in the averaging) and the weights for each pixel can be optimized according to a number of criteria such as the degree of smoothing. The above technique is akin to linear filtering since weighted averaging is equivalent to the convolution procedure in filtering.
While the second decimation technique described above is an improvement to the first decimation technique described above, in terms of the smoothness of the output image, the second decimation technique suffers from the problem described below with reference to FIG. 4 of the drawings. Referring to FIG. 4, a Bayer domain input image 24 is scaled down to an output image 26. The down scaling factor is 1:4 horizontally and vertically, and each output pixel in the output image 26 shown essentially covers an area of 4×4 pixels in the input image 24. Note that both G1 and G2 are green pixels. They are only labeled (named) differently because of traditional hardware arrangements in image sensors. The G1 and G2 pixels may go through different processing circuits in the image sensor and hence would have minor differences between them, but they are still pixels giving color information on the same primary color within the neighborhood. It will be appreciated that the weighted averaging in FIG. 4 is sub-optimal because, e.g., the G1 pixel 28 near the lower left corner of the input image is spatially much closer to the G2avg pixel 30 in the output image than to the G1avg pixel 32 in the output image.